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TITLE:  A  PERSPECTIVE  ON  COMMISSIONING  AND  EDUCATION— TOTAL  QUALITY,  TOTAL 
FORCE 

AUTHOR:  Dr.  Ronald  D.  Reed,  Lieutenant  Colonel,  USAF 

Internal  and  external  pressures  drive  leaders,  planners,  and  senior 
decision  makers  to  evaluate  educational  progreuns  for  efficiency,  effectiveness 
and  long-term  benefits .  This  paper  focuses  on  such  Issues  with  respect  to 
Department  of  Defense  (DOD)  commissioning  progreuns.  Meshed  with  older 
educational  concerns  for  development  and  reform,  a  growing  en^hasis  on  Total 
Quality  Management  (TQM)  offers  opportunities  and  challenges  in  meeting  such 
pressures  and  in  supporting  evaluation.  TQM  areas  of  particular  in^ortance  in 
tailoring  TQM  to  commissioning  programs  are  customer  and  product 
identification,  quality  definition  and  measurement,  leadership  and  teamwork  in 
organizational  culture,  and  benchmarking.  Data  is  needed  to  support  decision 
making  and  program  in^rovement  at  all  levels.  Several  metrics  of  con^arative 
quality  are  available,  with  one  being  surveys  of  supervisors  for  newly 
commissioned  officers.  A  case  is  made  that  better  coordination  of  evaluative 
data  and  commissioning  programs  is  needed.  Ultimately,  this  coordination 
should  extend  within  each  service,  across  the  joint  services,  and  through 
career-long  professional  military  education. 
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CHAPTER  I 

INTRODUCTION 


Objective  and  Scope 

Internal  and  external  pressures  drive  leaders,  planners,  and  senior 
decision  makers  to  evaluate  educational  programs  for  efficiency,  effectiveness, 
and  long-term  benefits.  This  paper  will  focus  on  such  issues  with  respect  to 
Department  of  Defense  (DOD)  commissioning  programs.  It  will  be  seen  that  Total 
Quality  Management  (TQM) ,  as  a  broad  philosophical  framework  with  an  emphasis  on 
practical  tools  for  decision  making,  poses  unique  opportunities  and  challenges 
to  support  such  evaluation.  A  key  factor  in  grasping  these  opportunities  and 
meeting  these  challenges  is  having  a  clear,  renewed  frame  of  reference  among 
commissioning  sources  reflecting  DOD  and  educational  TQM  initiatives.  Further, 
successful  evaluation  efforts  require  meaningful  information  to  support  decision 
makers  at  all  levels . 

The  fundamental  purpose  of  evaluation  in  TQM  is  inprovement,  and  this 
paper  will  examine  TQM  concepts  that  can  foster  inprovement  within  and  among 
military  services.  These  concepts  include  product  and  customer  identification, 
quality  definition  and  measurement,  leadership  and  teeunwork  in  organizational 
culture,  and  conpetition  or  benchmarking  from  a  commissioning  perspective.  It 
will  highlight  the  need  for  conparable,  comprehensive  evaluation  among 
commissioning  programs  with  an  understanding  that  these  programs  have  both 
common  and  unique  aspects  conpared  to  each  other  or  civilian  counterparts.  The 
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paper  will  also  summarize  aspects  of  program  evaluation  and  discuss  how  tools 
for  evaluation  have  the  potential  to  help  Integrate  commissioning  programs. 


In  focus,  this  paper  will  deal  primarily  with  the  US  Air  Force  Academy 
(USAFA)  as  it  relates  to  other  Air  Force  and  DOD  commissioning  sources;  total- 
force  concepts  will  extend  Issues  within  and  eunong  services.  The  paper  will 
further  focus  on  the  academic  environment  rather  than  administrative  or  support 
areas.  In  scope,  it  will  explore  some  current  issues  while  giving  a  basis  for 
further  discussions  by  decision  makers  and  workers  in  the  field.  Due  to  the 
con^lexity  of  the  issues,  it  must  remain  a  survey  in  character,  providing 
background  and  supplemental  appendices  without  becoming  a  primer  in  assessment, 
TQM,  or  other  areas.  In  the  end,  this  paper  will  have  moved  between  broad 
issues  and  specific  tools  in  an  exploration  of  ways  to  achieve  total  quality  in 
total-force  commissioning,  a  goal  needing  renewed  en^hasis  now. 

Background — Commissioning  and  Education 

Commissioning  within  DOD  centers  on  production  programs  from  three  primary 
services.  Army,  Navy,  and  Air  Force.  In  this  simplified  view  each  service  has 
three  main  avenues  available:  1)  short-term,  short-notice  training  through 
Officer  Candidate  School  (OCS),  Officer  Training  School  (OTS),  and  equivalent 
progreuns  in  which  military  training  is  intense  but  there  is  little  educational 
broadening;  2)  Reserve  Officer  Training  Corps  (ROTC)  progreuns  combining 
federally  funded  military  training  with  partially  funded  civilian  academic 
programs;  and  3)  four-year,  self-contained  service  academies  providing  a  full 
spectrum  of  academic  education  and  military  training  along  with  athletics  and 
other  broadening  programs.^  OCS/OTS  is  sometimes  seen  as  a  flexible,  readily 
expandable  source  of  officers  that  has  a  secondary  benefit  of  offering  an  avenue 
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to  comnisslonlng  by  enlisted  people.  ROTC  is  seen  as  another  flexible, 
expandable  source  of  cotomissioned  officers  with  broad  civilian  perspectives  and 
specific  skills,  as  well  as  being  an  ongoing  civil-military  link  in  our  society. 
The  service  academies  are  seen  as  a  controlled,  reliable  source  of  career 
officers  with  more  in-depth,  integrated  education  and  training  to  meet  long- 
range  national  needs  and,  in  many  eyes,  to  set  the  standards  of  military 
professionalism  in  their  services. ^ 

Exceptions  and  additions  to  this  sin^>le  view  are  many.  For  exanqple,  the 
Naval  Academy  provides  Marine  Corps  officers,  but  Marines  also  have  their  own 
training  programs,  as  do  other  uniformed  services.  Within  the  Air  Force  and 
other  services  there  are  added,  special  commissioning  programs  through  such 
avenues  as  the  Air  National  Guard  (the  Academy  of  Military  Science)  and  the 
medical  specialty  field  (Military  Indoctrination  for  Medical  Service  Officers).^ 
Further,  academy  cadets  may  choose  to  cross-commission  or  transfer  into  another 
service  branch,  as  may  some  ROTC  cadets.  Ultimately,  each  commissioning  program 
is  unique  in  scope  and  its  focus  on  service-  or  mission-specific  issues.  Still, 
cross-commissioning  and  the  common  purpose  of  commissioning  new  officers  imply 
that  although  there  is  program  uniqueness,  there  is  also  some  unity  of  goals, 
education,  and  training. 

Within  the  spectrum  of  education  and  training  given  by  commissioning 
programs,  education  per  se  involves  broader,  less  quantifiable  goals  linked  to 
the  mastery  of  academic  disciplines,  the  achievement  of  critical  cognitive 
capabilities,  and  the  evolution  of  social  or  other  values.  Commissioning 
training  has  been  seen  as  a  combination  of  "hard"  and  "soft"  (narrow  or  broad) 
aspects,  where  the  less  quantifiable  areas  in  broad  training  (of ficership,  etc.) 
blend  into  the  general  framework  of  education.^  In  evaluating  and  iit^roving 
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coRunissionlng  programs,  the  effectiveness  of  narrow  training  may  be  measured  by 
the  performance  of  selected  (often  psychomotor)  tas)cs,  but  the  effectiveness  of 
progr2uns  with  broader  goals  is  more  difficult  to  measure.  As  an  exfunple  of 
broad  goals,  USAFA  has  worked  since  1992  towards  a  mission  to  "develop  and 
inspire  air  and  space  leaders  with  vision  for  tomorrow."^  Con^arable  missions 
for  other  commissioning  programs  are  given  in  Appendix  I — they  set  a  range  of 
goals  affecting  progreun  evaluation  and  in^rovement .  Devising  and  in^lementing 
effective  evaluation  programs  becomes  more  difficult  as  goals  become  less 
tangible,  whether  within  each  commissioning  program  or  among  them. 

The  evaluation  and  integration  process  has  broader  implications  than  in 
iit^roving  individual  or  common  curricula.  It  relates  to  overall  force  structure 
and  our  ability  to  weigh  the  costs  and  benefits  of  commissioning  officers  from 
different  sources  within  and  among  the  services  or  between  commissioning  sources 
and  civilian  educational  institutions.  In  fact,  there  is  little  concrete  data  to 
con^are  the  quality  of  officers  from  each  source  or  to  foster  program 
in^rovement.  The  General  Accounting  Office  (GAO)  concluded  in  1991  that  the 
"services  have  done  relatively  little  research  to  formally  assess  the  quality  of 
officers  produced  through  the  various  commissioning  programs."®  The  Assistant 
Secretary  of  Defense  (Force  Management  and  Personnel)  was  forced  to  concur  in 
principle  with  this  criticism.^  In  exploring  TQM  as  a  focus  for  better 
evaluation,  leading  to  program  in^rovement  and  integration,  this  paper  will 
first  examine  the  context  of  pressures  for  such  evaluation  in  education  and 
commissioning. 
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CHAPTER  II 

PRESSURES  DRIVING  EVALUATION 

Several  factors  drive  a  desire  and  need  for  evaluation  of  the  educational 
process  and  Its  output,  factors  that  are  viewed  as  both  Internal  and  external  to 
the  educational  system.^  Such  factors  also  apply  to  the  commissioning 
environment  and  are  augmented  by  unique  factors  In  that  environment. 

Internal  Pressures 

Internally,  there  has  long  been  an  Institutional  and  Individual  desire  for 
lit^rovement  in  curricula  and  educational  technique.®  Bogue  and  Saunders  noted 
that  the  leadership  of  academia  has  always  asked  questions  of  purpose  or 
performance,  as  do  other  leaders. The  professionals  who  educate  do  the  same, 
as  reflected  in  movements  to  capitalize  on  new  educational  technologies  or 
debates  on  educational  philosophy  and  technique.  As  educators  work  to  perfect 
their  skills  and  art,  the  primary  focus  of  such  efforts  remains  on  the  student. 
Thus,  a  coupled  pressure  driving  educational  evaluation  has  been  the  desire  for 
student  development  towards  a  variety  of  goals.  Different  teachers  may  have 
different  views  of  the  proper  student- teacher  relationship  or  learning  model, 
yet  the  drive  is  fundamental  to  the  act  of  teaching. 

The  needs  and  desires  of  the  student  can  also  be  seen  as  an  Internal 
pressure  driving  evaluation  and  In^rovement  In  education.  These  needs  and 
desires  may  be  expressed  Individually  or  collectively  during  the  students ' 
participation  In  the  educational  system;  after  graduation  other  avenues,  such  as 
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alumni  associations,  can  aid  this  expression.  The  Influence  of  these  needs  and 
desires  on  the  educational  system  varies  over  time,  again  depending  on 
institutional  and  individual  views  of  the  student-teacher  relationship. 

Other  internal  pressures  are  administratively  driven,  such  as  the  need  by 
organizational  planners  to  make  con^arative  evaluations  and  resource  trade-offs 
among  departments  or  programs  in  times  of  decreased  resources .  Such  decreased 
resources  can  also  drive  educators  to  evaluate  their  programs  for  ways  to  be 
more  efficient  and  effective  by  modifying  teaching  techniques  to  deal  with 
decreased  budgets,  teacher  shortages,  and  other  constraints .  Much  can  be 
written  about  the  core  internal  pressures  for  educational  evaluation  and 
iit^rovement,  but  the  emphasis  in  recent  years  has  been  to  place  them  in  a  larger 
context  of  increasing  external  pressures  on  the  educational  system. 

External  Pressures 

Externally,  there  are  many  pressures  on  educational  institutions  for 
accountability  regarding  their  performance,  coupled  with  demands  for  increased 
oversight  and  management  access.  At  the  individual  level,  in  the  role  now  of  a 
potential  "customer, "  students  demand  an  education  responsive  to  individual 
needs  and  perspectives  at  a  reasonable  cost.^^  Parents,  taxpayers,  corporations, 
and  others  who  may  help  finance  education  demand  return  on  their  investment.^* 
Reinforcing  such  demands  from  the  public  sector  has  been  a  perceived  decline  in 
US  schools  and  the  capabilities  of  our  graduates  regarding  literacy  or  other 
critical  skills,  creative  thinking,  social  values,  and  the  relative  ability  to 
support  economic  development  and  adaptation  to  technological  or  other  change. 

Facing  such  demands  and  perceptions,  colleges  and  universities 
increasingly  must  compete  with  each  other,  as  well  as  with  corporate  and  foreign 
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educational  alternatives,  for  a  pool  of  new  and  returning  students. 
Commissioning  programs  also  must  con^ete  In  this  environment.  Since 

commissioning  programs  are  linked  to  overall  military  recruiting,  this  Is 

especially  evident  as  military  forces  are  reduced  and  military  careers  may  seem 
less  attractive.  Despite  a  prestige  that  attracts  a  surplus  of  candidates,  at 
USAFA  this  factor  has  been  augmented  by  such  negative  signals  as  decreased 

numbers  of  pilot  training  slots  and.  In  FY92,  legislation  that  decreased  the 
overall  class  size  and  attacked  traditional  incentives  such  as  the  granting  of  a 
regular  commission  to  graduates  (a  commission  favored  over  a  reserve  one  because 
of  increased  career  potential  or  security). It  can  be  expected  that  future 
officer  candidates  will  carefully  examine  the  value  of  commissioning  programs 
versus  other  education,  en^loyment,  and  national  service  opportunities. 

Added  to  such  "market-oriented”  external  pressures  for  accountability  are 
those  from  government  officials.  Increasingly,  local,  state,  and  federal 

governments  have  seen  the  oversight  of  education  as  a  legitimate  function 

requiring  decisions  based  on  educational  outcomes.^®  Central  to  oversight  are 
issues  of  responsiveness  to  the  needs  of  the  community,  standards  of  quality, 
and  the  cost-benefits  of  different  institutions  or  teaching  strategies  affecting 
the  investment  of  public  resources.  President  Johnson's  War  on  Poverty  was  seen 
by  some  as  a  watershed  in  focusing  attention  and  funds  on  education,  along  with 
demands  for  accountability  and  evaluation.^®  Similarly,  President  Bush  worked 
with  the  Department  of  Education  in  1991  to  reen^hasize  that  our  schools  must  be 
improved  and  must  develop  "an  en^hasis  on  accountability — setting  goals, 
objectively  measuring  progress  towards  these  goals,  changing  what  doesn't  work, 
and  rewarding  what  does."^°  The  new  administration  can  be  expected  to  continue 
this  trend  towards  accountability  linked  to  evaluation. 
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Related  to  such  Issues  are  cost-related  concerns  that  intensify  with 
decreasing  resources.  Cost  analysis  is  a  "policy-oriented  dimension  of 
evaluation"  to  help  policy  makers  allocate  resources  to  maximize  outcome  for 
given  input. Cost-effectiveness  and  cost-benefit  analyses  derived  from  public 
sector  initiatives.  Thus,  cost-benefit  analysis  (which  can  Include  broader, 
long-term  effects  of  a  program)  was  developed  in  the  1930s  to  help  Congress 
evaluate  water-resource  projects  proposed  for  the  Army  Corps  of  Engineers. 
Cost-effectiveness  analysis  was  developed  for  DOD  in  the  1950s  to  help  analyze 
the  relative  merits  of  weapons  systems,  roles  and  missions. Levin  has  noted 
that  difficulties  in  such  cost-related  studies  include  the  need  to  quantify 
multiple,  often  vague,  objectives  as  well  as  the  requirement  to  deal  with 
imperfect  information  and  a  lack  of  understanding  among  policy  makers  about  the 
method. 23  Even  so,  cost  studies  now  are  fundamental  in  accountability  efforts. 

Such  issues  of  government  accountability  have  a  strong  impact  on 
commissioning  programs.  For  exan^le,  cost-benefit  issues  have  been  one  factor 
in  the  Congressional  review  of  officer  accessions  and  in  decisions  regarding  the 
size,  structure,  or  even  existence  of  various  commissioning  sources.  DOD  spends 
over  $1  billion  annually  for  commissioning  programs,  a  sum  earning  growing 
Congressional  oversight. 2^  The  service  academies  are  seen  as  the  most  expensive 
commissioning  sources,  with  estimated  costs  per  graduate  ranging  from  about 
$153-228,000  at  the  academies  con^ared  to  $53-58,000  for  ROTC  scholarships  and 
$15-$20,000  for  OCS/OTS  training. 23  USAFA  and  West  Point,  in  particular,  were 
scrutinized  by  the  GAO  in  1991  regarding  costs  per  cadet  and  the  benefits  of 
that  cost. 26  Subsequently,  the  Secretary  of  the  Air  Force  recommitted  USAFA  to 
being  a  "cost  effective  source  of  dedicated,  high  quality  officers  for  the 
nation.  "2"^  As  USAFA  and  other  commissioning  sources  seek  to  reaffirm  their 
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merits  to  senior  decision  makers,  legislative  trends  for  the  time  being  retain 
diversification  of  investment  among  commissioning  sources  and  their  unique 
capabilities  and  benefits  (whether  functional  or  political) .2® 

The  issue  of  cost-benefits  is  not  the  only  Congressional  issue  with 
commissioning  programs,  just  as  Congress  has  expressed  interest  in  broad- ranging 
civilian  processes  or  outcomes .  Other  issues  include  the  cultures  of  the 
commissioning  programs  and,  indirectly,  their  success  at  developing  less 
tangible  social  or  leadership  characteristics — issues  such  as  hazing  and  sexual 
or  racial  harassment. Additionally,  there  can  be  curriculum  concerns,  such  as 
the  amount  of  joint  service  education  received  in  an  increasingly  joint- 
operations  world.  In  FY92  and  FY93  Congress  questioned  the  merits  of  the  USAFA 
educational  model  of  an  all-military  faculty  cort^ared  to  the  more  civllianlzed 
model  at  Annapolis  or  to  civilian  schools  in  general.^®  Congress  also  directed 
the  Secretary  of  Defense  to  establish  uniform  reporting  and  oversight  procedures 
for  the  academies  and  to  determine  where  they  fit  in  the  command  structure. 
Reports  from  the  Secretary  of  Defense  to  Congress  are  due  this  year. 

Whether  higher  DOD  and  service  headquarters  should  be  considered  internal 
or  external  pressures  depends  on  one's  perspective.  Although  a  corporate  vision 
is  needed  for  program  integration,  from  the  view  of  planners  within  individual 
programs  these  may  appear  to  be  external.  Current  pressures  for  evaluation  at 
the  DOD  level  have  centered  on  the  Directorate  for  Accession  Policy.  DOD  holds 
coordination  meetings,  such  as  one  slated  for  May  1993  among  ROTC  programs.  It 
also  considers  some  initiatives,  such  as  one  considering  joint  training  for  the 
first  two  years  of  all  four-year  ROTC  programs.  will  be  seen,  most 

coordination  and  evaluation  has  otherwise  been  left  to  the  services .  Within  the 
Air  Force,  reorganization  efforts  focused  in  1992  on  The  Year  of  Education  and 
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Training.  One  headquarters  initiative  was  an  idea  of  placing  all  commissioning 
sources  under  the  newly  formed  Education  and  Training  Command,  an  idea  echoed  in 
Senate  Armed  Services  Committee  discussions.^^  This  initiative  was  not 
inplemented  for  USAFA,  but  the  potential  remains  for  a  single  command-level 
manager  to  weigh  differences  in  the  educational  programs,  goals,  and  outcomes  of 
all  commissioning  sources. 

Another  pressure  driving  evaluation  of  commissioning  programs  is 
represented  by  the  organizations  and  supervisors  to  which  commissioned  officers 
are  assigned.  These  units  set  limits  on  the  acceptability  of  officers  in 
certain  categories  or  of  supporting  educational  programs  (i.e.,  qualifications 
for  medical  or  civil  engineering  fields),  may  take  part  in  hand-picking  specific 
officer  candidates  for  inclusion  in  their  career  fields,  and  may  provide 
feedback  based  on  their  impressions  of  the  officers  they  receive.  Commissioning 
programs  have  a  direct  responsibility  to  fill  the  needs  of  the  units  that 
receive  their  graduates.  This  pressure  will  be  discussed  more  later  regarding 
customer  relations  of  civilian  and  commissioning  educational  programs  under  TQM. 

Finally,  Individual  officers,  in  parallel  to  their  civilian  counterparts, 
may  provide  feedback  upon  graduation  and  throughout  their  careers  that  lends 
information  for  program  in^rovement.  In  helping  to  build  officership  and  career 
motivation  into  their  graduates,  commissioning  programs  hope  to  foster  qualities 
that  will  last  throughout  a  career  and  beyond.  Thus,  the  opinions  of  graduates 
become  a  strong  pressure  regarding  program  success  and  direction. 

All  such  factors,  internal  and  external,  exert  pressure  on  commissioning 
sources  for  continuous  evaluation  coupled  to  a  program  of  organizational 
introspection,  iitprovement,  and  change.  TQM,  as  seen  in  its  growing  integration 
into  US  management  practices,  is  a  way  to  focus  such  evaluation. 
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CHAPTER  III 

TQM— FOUNDATIONS  IN  DOD  AND  EDUCATION 

Along  with  earlier  pressures,  the  use  of  TQM  philosophy  and  practices  DOD- 
wlde  can  be  seen  as  another  factor  driving  evaluation  In  commissioning  and 
education  programs.  This  factor  can  be  considered  an  external  one,  as  It  Is 
advocated  from  senior  business  and  government  levels .  It  Is  also  an  Internal 
one,  as  It  Is  accepted  and  applied  at  the  organizational  level.  With  elements 
of  both  Internal  and  external  pressures,  then,  TQM  helps  merge  the  demands  of 
these  pressures  for  Improved  evaluation  and  Integration.  Concurrently,  It  helps 
merge  management  activities  seen  both  In  DOD  and  civilian  education. 

National  and  Government  Directions  in  TQM 

TQM  is  becoming  a  widely  adopted  national  way  of  pursuing  business.  This 
adoption  resulted  mainly  from  the  success  of  TQM  In  Japan,  where  Its  foundations 
were  laid  by  American  advocates  such  as  W.  Edwards  Demlng  and  Joseph  M.  Juran  In 
the  early  1950s  and  were  built  on  by  such  Japanese  experts  as  Kaoru  Ishlkawa.^^ 
In  contrast,  US  con^anles  for  many  years  focused  on  short-term  profits,  assumed 
tradeoffs  between  quality  and  cost,  and  secure  markets  for  their  products  backed 
by  management  based  on  "bottom  line”  figures,  mobile  managers,  and  rating  of 
employees  In  meeting  set  goals. Now,  with  Increasing  global  con^etltion,  US 
companies  are  turning  to  "quality"  as  a  way  to  In^rove  processes  and  products. 

Key  concepts  of  TQM  are  a  development  of  long-term  organizational  vision, 
commitment  to  continuous  in^rovement  (the  Japanese  concept  of  kaizen) ,  a  focus 
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on  customer  needs  and  satisfaction,  benchmarking  or  con^arison  against  the  best 
in  the  field,  eliminating  coercive  management  tools  and  fear  among  employees, 
and  the  training  and  eitpowerment  of  employees  to  do  their  best  work.  Many  TC2M 
concepts  build  on  prior  management  principles  of  decentralized,  participatory 
management . TQM  consultants  and  analysts  have  now  enumerated  such  concepts 
for  widespread  use,  concepts  that  are  applicable  here  (J^pendices  II-VI)  . 

Milakovich  has  pointed  out  that  government  organizations  mirrored  many 
faults  of  US  business  after  World  War  II. Yet,  governments  faced  pressures 
similar  to  business:  decreasing  resources,  a  need  to  improve  productivity,  lack 
of  customer  satisfaction,  trouble  retaining  qualified  workers,  and  interagency 
or  global  competition.  Now  the  focus  on  TQM  seen  in  business,  health  care,  or 
other  organizations  is  mirrored  in  government.^®  In  1989  President  Bush  stated, 
"Reasserting  our  leadership  position  will  require  a  firm  commitment  to  Total 
Quality  Management  and  the  principle  of  continuous  inprovement.  .  .Quality 
improvement  principles  apply.  .  .to  the  public  sector  as  well  as  private 
enterprise."®®  The  US  Federal  Total  Quality  Management  Handbook  now  defines 
quality  as  "meeting  the  customer's  rex,uirements,  needs  and  expectations,  the 
first  time  and  every  time"  and  outlines  principles  of  government-wide  TQM  use.^° 

Difficulties  do  exist  in  inplementing  TQM  principles  in  the  government. 
For  one  thing,  politicians  often  focus  on  near-term  political  rewards  when 
distributing  national  resources.  Politicians  also  must  deal  with  coitplex, 
conflicting  goals  and  constituencies,  the  realities  of  compromise,  and  the  need 
to  evaluate  programs  whose  success  or  failure  is  subject  to  multiple 
interpretations.^®  Bureaucracies  can  resist  change  and,  further,  the  tenure  of 
senior  leaders  may  be  short  (not  just  in  Congress,  but  averaging  only  24  months 
for  an  assistant  secretary  in  the  federal  executive  department),  thus  degrading 
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constancy  of  vision  and  support.  Despite  such  factors,  many  organizations 

within  local,  state,  and  federal  governments  have  applied  principles  to 

their  operations  and  benefited  by  doing  so.^^ 

Likewise,  DOD  has  worked  to  implement  TQM,  making  it  DOO's  official 

management  ingirovement  system  in  1988.'*^  This  movement  overlays  earlier  DOD 

efforts  for  quality  "control”  via  other  management  systems  and  helps  respond  to 

decreasing  budgets,  rapid  technological  change,  and  a  need  to  reexauidne 

customers  and  missions  in  a  shifting  global  paradigm. As  applied  in  DOD: 

Total  Quality  Management  (TQM)  is  both  a  philosophy  and  a  set  of 
guiding  principles  that  represent  the  foundation  of  a  continuously 
ing>rovlng  organization.  TQM  is  the  application  of  quantitative 
methods  and  human  resources  to  in^rove  the  material  and  services 
supplied  to  an  organization,  all  the  processes  within  an 
organization,  and  the  degree  to  which  the  needs  of  the  customer  are 
met,  now  and  in  the  future.  TQM  integrates  fundcuxiental  management 
techniques,  existing  in^rovement  efforts,  and  technical  tools  under 
a  disciplined  approach  focused  on  continuous  in^rovement.^® 

DOD-wide  in^lementation  of  TQM  makes  its  integration  into  commissioning  programs 

in  accord  with  other  DOD  activity,  as  does  its  current  application  to  education. 


TQM  Linkages  to  Education 

TQM's  adaptation  and  application  to  the  educational  system  is  still 
evolving  from  early,  variable  efforts  as  more  institutions  build  and  con^are 
progrcuns.^®  Many  TQM  concepts  work  synergistically  with  such  prior  internal 
educational  efforts  as  curriculum  and  faculty  development  or  assessment.  Others 
have  been  applied  in  administrative  and  infrastruture  areas  to  in^rove  service 
and  support  of  educational  processes.  Yet,  inqplementation  raises  unique  issues. 

Among  these  issues  is  that  of  defining  the  product  and  customer,  for  in 
TQM  the  needs  of  internal  and  external  organizational  customers  must  be  met. 
White  argues  that  both  students  and  en^loyers  of  graduating  students  are 
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customers,  of  which  the  most  vital  is  the  student  (with  student  skills  setting  a 
demand  for  future  university  services  and  graduates).  He  feels  that  academia's 
key  product  is  the  education  it  provides. From  another  perspective,  students 
(in  gaining  an  education  and  then  representing  the  educational  process)  have 
been  seen  as  both  customers  and  products — in  both  capacities  defining  quality 
for  an  institution.^®  Going  further,  in  actively  participating  to  acquire  their 
own  education  students  become  at  one  time  part  of  the  process,  the  chief 
products,  and  primary  customers.  Parents  have  been  equated  to  customers  or  as 
an  "educational  equivalent  of  stockholders."^®  Other  external  customers  may 
include  local  communities,  grantors,  industry,  legislatures,  boards  of 
education,  other  educational  institutions,  alumni,  and  the  nation  in  abstract.®® 

The  issue  of  product  and  customer  identification  is  con^licated  by  other 
goals  seen  in  educational  organizations  beyond  teaching  well,  such  as  community 
activities  and  faculty  scholarship  or  research.®^  A  growth  of  research-centered 
universities  has,  for  some,  made  research  prestige  a  major  goal,  research 
productivity  a  major  output,  and  research  grants  a  major  input  to  solvency. 
White  and  others  argue  that  research  needs  of  professors  are  less  inportant  than 
academic  needs  of  students  and  that  the  process  of  research  in  education  may  be 
more  in^ortant  than  its  results.  This  argues  against  any  academic  culture  in 
which  professors  see  the  institution  as  existing  to  serve  their  goals  rather  one 
in  which  they  function  to  provide  a  service  to  students. ®2  Still,  in  such  cases 
as  land-grant  colleges,  research  or  other  goals  may  be  part  of  the  mission.®® 
Tension  remains  as  the  relationships  of  all  internal  and  external  suppliers  and 
customers  in  a  TQM  framework  evolve  and  are  debated  in  academia. 

Another  cited  issue  is  the  inpression  that  TQM  infringes  on  traditional 
cultural  norms  found  in  academia,  a  model  typified  by  departmental  autonomy. 
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individual  diversity,  academic  freedom,  and  a  professional  elitism  or 
disciplinary  specialization  that  resist  change. Such  characteristics,  in  a 
narrow  sense,  run  counter  to  teamwork  concepts  in  TQM.  Removing  these  barriers 
or  placing  them  in  a  cooperative  context  is  one  educational  reform  that  is  part 
of  the  TQM  movement.  Thus,  support  for  individuality  and  academic  freedom  need 
not  endorse  poor  teaching,  nor  do  department  autonomy  and  decentralized  program 
execution  bestow  the  right  to  ignore  broader  goals  of  a  core  curriculum. 

A  related  barrier  is  suboptimization  or  parochialism  2unong  departments  and 
caucuses  due  to  disciplinary  and  program  loyalty  or  con^etition  for  students  and 
funding. The  degree  to  which  this  effect  is  counterproductive  requires  a 
vision  of  total  mission.  Juran  has  stressed  that  top  leaders  must  recognize 
"all  major  quality  problems  are  Interdepartmental"  and  pursuit  of  local  goals 
can  undermine  overall  effectiveness.^^  Likewise,  Deming  urges  intracompany 
cooperation  to  maximize  benefits  to  the  organization  and  its  customers  rather 
than  setting  up  divisions  or  departments  as  con^etitive  profit  centers. The 
same  can  be  said  in  academia.  Successful  programs  do  exist  to  restructure  and 
integrate  academic  programs.  One  well-known  effort,  begun  in  1973  by  Alverno 
College,  fosters  curriculum-wide  abilities  in  communication,  analysis,  problem 
solving,  valuing,  social  interaction,  involvement,  environmental  responsibility, 
and  aesthetic  response.  Such  programs  must  ensure  "the  curriculum  is  a 
collaborative  effort  which  transcends  departments  and  divisional  structures."®® 
One  technique  to  build  such  unity  of  effort  is  by  forming  quality  circles  (joint 
working  groups)  to  pool  knowledge,  effort,  goals,  and  commitment.®® 

Variation  is  another  issue.  Reducing  process  variation  to  achieve  product 
consistency  under  TQM  might  seem  appropriate  in  business  but  not  in  producing 
works  of  art  or  creative  individuals.  This  has  caused  some  educators  to  balk  at 
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a  production  metaphor  for  education  rather  than  a  transactional  or  "handmade” 
one.®°  There  are  Inherent  differences  In  students,  faculty,  or  Infrastructure — 
a  common  variation  In  the  educational  process,  but  a  variety  or  diversity  that 
Is  actively  sought.  Flexible,  tailored  curricula  for  Individualized  learning, 
augmented  by  con^uterlzed  Instruction,  can  help  cope  with  and  build  on  this 
diversity  rather  than  eliminating  It.^^  Where  reducing  variation  may  apply  In 
academia  Is  In  ensuring  that  standards  of  achievement  and  teaching  quality  are 
set,  and  that  all  students  and  teachers  are  Inspired  and  assisted  to  reach  them. 
Reducing  variation  can  mean  ensuring  that  prerequisites  for  one  course  are 
reliably  taught  In  preceding  ones.®^  Reducing  variation  for  broad  goals  such  as 
creativity  or  critical  thinking  does  not  Imply  a  rigidity  of  mind,  only  that 
students  reach  a  level  of  capability  In  these  areas — a  level  enabling  them  to 
express  those  capabilities  with  Individuality.  Any  externally  applied  standards 
for  accountability  also  should  fit  within  this  framework.®^ 

A  final,  fundeunental  Issue  Is  defining  and  measuring  quality  or  academic 
excellence  In  programs  with  broad  educational  goals. One  early  problem  seen 
by  developers  of  educational  evaluation  systems  was  that  such  "programs  have 
many  alms — often  as  many  alms  as  there  are  stockholders.  Progreuns  have  fuzzy 
borders,  so  that  It  Is  not  always  plain  sailing  to  determine  precisely  what  the 
program  was  that  required  evaluation."®®  Linked  to  difficulties  In  defining 
progreuns  Is  gauging  their  success  through  measurement  and  analysis,  which  are 
critical  to  TQM  and  Its  support  of  continuous  In^rovement.  Intangible  endpoints 
are  hard  to  measure,  yet  both  traditional  educational  assessment  and  TQM  are 
data-driven.®®  It  Is  fortunate  that  academia  has  a  well-established  emphasis 
on  evaluating  and  Improving  Its  programs.  This  foundation  has  many  linkages  and 
synergies  with  TQM,  with  one  exan^le  given  In  ^^pendlx  VII.  Yet,  educational 
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evaluation  as  a  field  of  disciplined  study  only  began  this  century. Metrics 
are  still  evolving,  and  In  this  evolution  several  distinctions  must  be  made. 

First,  will  data  be  related  to  process  Input  or  output?  For  example, 
progreun  costs  are  widely  sought  Input  data.  However,  costs  are  only  part  of  a 
larger  picture,  and  they  may  not  be  easily  definable  or  comparable,  especially 
when  there  is  no  uniform  reporting  guidance.®®  Other  Input  data  can  include 
hours  spent  in  class,  teacher-student  ratios,  length  of  readings,  or  credentials 
and  training  of  teachers.  Assessing  input  data  assumes  Its  correlation  to 
ultimate  output,  but  care  must  taken  that  institutional  and  learning  processes 
support  such  a  linkage.  Thus,  the  number  of  PhDs  on  a  faculty  has  less  meaning 
If  they  teach  poorly  or  If  those  actually  teaching  in  class  are  only  graduate 
assistants.  Output  data  can  also  be  diverse,  including  test  scores,  the  number 
of  graduates  or  degrees  obtained,  national  honors,  survey  results,  and  student 
portfolios  of  creative  works.  Various  sources  of  data  are  in  Appendix  VIII. 

Second,  will  data  be  subjective  or  objective,  qualitative  or  quantitative? 
TQM  emphasizes  quantitative  data,  but  care  must  be  taken  to  avoid  measuring  what 
is  easy  (e.g.,  number  of  lessons  or  readings,  length  of  readings,  faculty 
credentials,  curriculum  content,  test  scores,  contact  time)  Instead  of  seeking 
measures  of  real  performance  relevant  to  goals  and  analyzing  processes  to 
achieve  those  goals.  TQM  principles  also  identify  quality  as  what  is  perceived 
as  such  by  the  customer (s),  necessitating  some  subjective  evaluation. 
Subjective  data  increases  the  flexibility  and  scope  of  evaluation  and  can 
"provide  richer  and  more  relevant  information"  than  purely  quantitative  data, 
but  it  is  susceptible  to  bias  and  may  be  viewed  skeptically.®®  One  can 
"quantify"  subjective  data,  as  shown  in  systems  analysis  and  TQM  metrics,  but  in 
converting  subjective  assessments  to  quantifiable  data  (as  in  building  rating 
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scales  for  surveys  or  coarse  critiques)  care  must  be  taken  in  considering  the 
appropriateness  of  the  conversion  along  with  associated  biases  or  assun^tions . 

Third,  who  should  be  setting  criteria  by  which  to  evaluate  academic 
performance,  internal  or  external  sources?  Self-assessment  advocates  note  that 
the  most  meaningful,  most  useful  criteria  are  internally  generated. Still, 
some  criteria,  such  as  cost  criteria,  may  be  externally  driven.  Further,  even 
if  criteria  are  internally  generated,  they  may  be  externally  reviewed — often 
accreditation  or  other  program  review  organizations  build  their  evaluation  based 
on  internally  generated  goals  criteria.  This  basic  question  of  who  requires, 
acquires,  and  acts  on  data  also  relates  to  its  ultimate  function. 

Fourth,  then,  will  data  be  for  summative  or  formative  purposes?  These 
terms  were  first  used  in  1967  to  distinguish  two  roles  of  evaluation:  formative 
evaluation  supports  process  in^jrovement;  summative  evaluation  serves  any  other 
purpose,  whether  to  support  policy  decisions  or  to  provide  data  for  research.’’^ 
Normally,  formative  evaluation  is  Internal  and  summative  is  external,  but  this 
is  not  always  true.  Changes  in  classification  can  occur,  as  when  a  summative 
evaluation  spurs  in^jrovement  or  when  a  formative  evaluation  is  distributed  and 
used  for  summative  purposes — it  is  a  question  of  context. The  focus  in  TQM  is 
on  formative  evaluation.  Summative  evaluation,  in  contrast,  can  spark  counter¬ 
productive  anxiety  in  an  organization;  yet  it  is  a  reality  in  education  and 
government,  a  reality  linked  to  pressures  for  accountability  seen  earlier. 

All  forms  of  evaluation,  all  metrics  and  data,  can  play  a  part  in  TQM. 
The  challenge  of  evaluation  in  education  is  much  the  same  as  in  any  TQM 
endeavor.  That  challenge  is  to  find  meaningful  measures  of  merit  relating  to 
program  goals,  measures  that  can  be  understood  by  decision  makers  and  can  be 
used  primarily  for  program  improvement. 
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CHAPTER  IV 


TQM  AND  COMMISSIONING— ISSUES,  CHALLENGES,  AND  OPPORTUNITIES 


TQM  and  The  Culture  of  Commissioning  Sources 

At  the  crossroads  of  DOD  and  academia,  commissioning  programs  can  apply 
TQM  as  a  freune  of  reference  to  deal  with  both  internal  and  external  pressures 
for  evaluation.  Indeed,  these  progreuns  have  already  included  TQM  principles  and 
practices  in  their  operations.  For  exan^le,  in  1991-92  USAFA  embarked  on  a 
broad  review  of  its  curriculum  and  teaching  techniques  in  an  effort  linked  to 
TQM  application.^^  The  Naval  Academy  also  has  a  strategic  plan  under  the 
banners  of  Total  Quality  Education  (TQE)  and  Total  Quality  Leadership  (TQL) 
Such  efforts  do  not  mean  that  TQM  can  be  applied  without  taj.loring  it  to  the 
commissioning  environment.  That  the  culture  and  goals  of  military  commissioning 
progr<uxis  are  different  from  academic  counterparts  provides  the  first  of  many 
issues,  challenges,  and  opportunities. 

As  noted  earlier,  commissioning  programs  are  a  combination  of  education 
and  training,  with  the  enphasis  varying  among  commissioning  sources.  Academies 
are  able  to  seek  broad  cross-curriculum  synergy  between  education  and  training; 
other  commissioning  sources  are  limited  by  time  and  resources .  All  civil  and 
military  educational  systems,  though,  face  a  similar  problem  in  defining 
quality.  This  is  especially  true  when  long-term  benefits  of  education  are 
examined  rather  than  short-term  knowledge  or  skills.  Voorhees  has  noted  that 
the  problems  in  academia  of  measuring  the  "success,  failure,  or  adequacy  of 
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investment"  in  achieving  academic  excellence  parallel  problems  of  the  military 
in  measuring  its  success  at  achieving  national  security,  a  parallel  extended  by 
their  public  funding  and  accountability  to  the  nation. As  such,  commissioning 
sources  are  again  at  the  crossroads — one  must  judge  both  their  ability  to 
educate  and  their  long-term  contribution  to  national  defense. 

A  recent  issue  facing  USAFA  is  the  role  and  relative  merits  of  civilian 
and  military  faculties,  an  issue  not  faced  by  academia.  Conflicting  goals  exist 
regarding  the  desire  to  give  students  professional  military  contacts  and  current 
insights  (favoring  military  faculty  on  a  rotating  basis)  and  long-term  expertise 
and  commitment  in  academics  (favoring  stable  military  or  civilian  faculty) 

In  1992,  the  Air  Force  followed  Congressional  guidance  to  begin  civilianizing 
its  faculty  to  the  50%  level.  This  will  change  an  organizational  culture 
represented  in  past  USAFA  guiding  principles  as  a  "community  of  soldier- 
scholars."^®  Despite  concerns  that  this  move  will  damage  USAFA' s  goals  and 
values,"'®  this  culture  must  now  adapt  not  only  to  con^ly  with  Congressional 
intent  but  also  to  build  a  new  teamworlc  for  in^rovement  in  consonance  with  TQM. 

Further  regarding  the  cultures  of  academia  and  military,  some  authors  have 
felt  that  typical  academic  independence  and  lac)c  of  leadership  training  may  be 
at  odds  with  the  strong  culture,  leadership,  and  teamwor)c  vital  to  TQM.®°  Here 
the  military  may  be  ahead  in  training  and  temperament,  though  many  colleges  are 
disproving  such  stereotypes.  Certainly,  participatory  leadership  found  in 
academia  may  be  quite  applicable  to  TQM;  and  autocratic  or  adversarial  styles, 
whether  in  academia  or  elsewhere,  can  damage  the  learning  and  TQM  environment.®^ 

The  unique  culture  and  missions  of  commissioning  sources  affect  their 
con^arability  and  limit  application  of  TQM-linJced  concepts  of  con^etition  and 
benchmarlcing.  Con^aring  service  academies  to  civilian  institutions  (the  "free 
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market"  of  academia)  has  limitations.  As  noted,  commissioning  sources  con^ete 
against  civilian  institutions  for  students;  but  only  USAFA  competes  on  the 
equivalent  basis  of  offering  a  baccalaureate  degree.  Even  so,  academies  are  not 
research  oriented  and  their  mission  includes  other  programs  interwoven  with 
academics  that  take  cadet  time.®^  If  other  commissioning  sources  compete  with 
civilian  institutions,  it  is  more  in  the  area  of  offering  career  opportunities. 
With  or  without  direct  con^etition,  though,  benchmarking  remains  a  key  to 
in^rovement  by  seeking  models  of  excellence  and  learning  from  them.®^  In  this 
context,  it  is  doubtful  that  any  single  civilian  institution  can  act  as  a  model 
for  the  USAFA  process.  Certainly,  there  are  models  (such  as  Alverno)  for 
integrating  key  concepts  or  values  across  a  curriculum.  There  are  also  superb 
disciplinary  models  for  academic  departments,  though  models  with  narrower  goals 
and  greater  resources.  There  are  vocational  or  other  models  that  integrate 
education  and  training.  All  these  may  be  of  use.  A  more  sensitive  question  is 
how  commissioning  sources  should  view  each  other  in  terms  of  conpetition  and 
benchmarking.  This  will  be  discussed  later  regarding  total-force  commissioning. 

Multiple  Customers  and  Requirements,  Products  and  Responsibilities 

As  seen,  several  authors  cite  students  as  the  primary  customers  of  the 
educational  system,  as  well  as  part  of  the  process  and  a  key  product.  As  in 
academia,  external  pressures  on  commissioning  programs  can  also  represent 
customers.  Depending  on  one's  vision,  an  extended  list  of  such  customers  may 
include;  the  nation  as  a  whole.  Congress  and  other  agencies  representing  the 
government  or  taxpayers,  the  military  services,  initial  and  follow-on  units  to 
which  graduates  are  assigned,  students  or  graduates,  foreign  students  or 
graduates  and  their  governments  (with  whom  USAFA,  for  exan^le,  provides  one 
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avenue  of  politico-military  contact),  and  even  planners  of  career-long 
professional  military  education  such  as  Squadron  Officer  School. 

Commissioning  programs  may  be  ahead  of  academia  in  feeling  the  immediacy 
of  their  many  customers  and  focusing  on  their  needs.  Certainly,  these  programs 
were  founded  on  service  to  country  and  are  accustomed  to  both  military  and 
civilian  oversight.  Another  reason  these  programs  may  be  more  focused  on  the 
customer  is  that  officers  in  commissioning  programs  )cnow  they  are  helping  build 
not  only  graduates,  but  colleagues  in  a  profession  and  perhaps  combat. These 
instructors  or  planners  are  thus  representatives  of  a  ]cey  external  customer. 
This  situation  may  be  a  parallel  to  civilian  education  in  specialty  fields  or 
preparation  for  advanced  degree  training  and  professional  service.  Such 
cultural  issues  con^llcate  producer-product  relationships,  though  one  still 
cannot  submerge  the  classic  educational  needs  of  the  student. 

At  the  same  time,  just  as  universities  have  secondary  products,  so  do 
commissioning  sources.  Service  academies  in  particular  may  engage  in  research 
and  consultation,  functions  not  detailed  in  formal  mission  statements  though 
they  may  be  supported  with  funds  and  such  facilities  as  the  Seiler  Research 
Laboratory  collocated  at  USAFA.  The  mission  of  USAFA  remains  rightfully  focused 
on  its  cadets,  but  the  faculty  job  description  used  for  Officer  Performance 
Reports  also  eit^hasizes  consultation  and  research  for  the  Air  Force  and  DOD. 
This  in^jlies  a  broader  vision  of  USAFA,  as  an  ongoing  resource  of  expertise  in 
addition  to  a  source  of  commissioned  graduates.  Beyond  this,  military 
supervisors  at  commissioning  sources  are  committed  to  the  professional 
development  of  their  officers.  This  has  led  some,  at  least  at  USAFA,  to  refer 
to  departing  military  instructors  as  the  "second"  graduating  class  each  year — 
another  set  of  customers  or  products  on  active  duty,  another  set  of  officers  the 
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Academy  must  serve  and  by  which  it  is  judged.  Again,  priorities  for  secondary 
products  or  missions  must  be  balanced  within  an  overall  organizational  vision, 
an  area  where  problems  facing  service  academies  parallel  those  facing  academia. 


Evaluation  and  Total-Force  Commissioning 

Having  seen  educational  and  TQM  linkages  to  commissioning  programs,  what 
are  the  needs  and  prospects  for  integrating  these  programs  in  a  TQM  framework? 
Within  the  services.  Air  Force  programs  are  coordinated  at  the  Air  Staff  level. 
A  Commissioning  Education  Memorandum  of  Understanding  (CEMU)  among  commissioning 
sources  sets  common  education  and  training  goals  focused  on  of ficership. A 
Commissioning  Education  Committee  (CEC)  meets  biennially  to  revise  the  CEMU  and 
support  cooperative  research  among  the  progreuns .  The  CEC  focuses  on  broad 
goals,  not  metrics;  and  program  evaluation  has  been  left  to  each  commissioning 
source.®®  The  Army  has  had  a  Lieutenant  Accessions  Requirements  Working  Group 
since  1988-89  to  provide  some  similar  coordination.®’’  According  to  other- 
service  and  DOD  counterparts,  in  late  1992  the  Navy  reduced  Pentagon-level 
coordination  efforts  and  moved  or  eliminated  comparable  points  of  contact.®® 

There  has  been  less  active  management  or  oversight  by  DOD  offices — this 
has  historically  been  left  to  the  services  as  they  organize,  train,  and  equip 
forces.®®  In  the  past  a  DOD  Committee  on  Excellence  in  Education  helped 
establish  officer-exchange  programs  cunong  service  academies.®®  In  response  to 
GAO  criticism,  in  1992  the  previously  cited  Directorate  for  Accession  Policy, 
Office  of  the  Assistant  Secretary  of  Defense  (Force  Management  and  Personnel) 
was  identified  to  Congress  as  DOD's  focal  point  in  this  matter.  ®’^  Still,  even 
within  this  context,  service  programs  build  on  their  unique  capabilities  along 
with  the  time  and  resources  available  to  produce  commissioned  officers. 
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If  commissioning  is  seen  as  one  corporate  process  within  DOD,  then 
integrated  management  makes  sense  even  if  execution  is  decentralized. 
Alternately,  lacking  strong  oversight,  should  commissioning  progreuns  be  viewed 
as  separate  entitles  that  produce  a  similar  product?  If  so,  in  the  business 
world  they  would  be  seen  as  con^etitive  processes  in  some  sense.  Among 
themselves,  military  services  maintain  that  commissioning  programs  are  not  in 
con^etition  or  truly  con^arable,  but  rather  have  conqplementary  purposes  and  make 
unique  contributions  to  producing  officers.^^  This  position  is  correct  in  many 
ways;  it  also  avoids  interservice  and  intraservice  rivalry.  Still,  some  form  of 
con^etition  can  spur  and  guide  Improvement.  Elsewhere  in  DOD,  part  of  TQM's 
application  is  the  measurement  of  performance  and  costs  to  enable  comparisons 
between  products  and  to  aid  in  iir^rovement  (as  in  logistics  support  centers)  . 
Further,  as  GAO  and  Congressional  action  have  shown,  commissioning  programs  will 
be  con^ared,  in  a  summative  external  sense  if  not  a  formative  internal  one. 

It  should  be  noted  that  even  if  commissioning  sources  were  viewed  as 
competing  organizations.  Denting  calls  for  cooperation.  A  business  analogy  is  in 
setting  common  standards  with  the  customer  so  continuous  improvement  can  be 
focused  better.®^  Another  analogy  that  might  be  applied  here  is  the  defense 
industry,  where  true  con^etition  does  not  exist — it  more  closely  resembles  a 
monopsony-monopoly  relationship.^^  Similarly,  DOD  is  the  only  customer  for 
commissioning  programs,  and  there  is  no  free  entry  into  con^etition  to  produce 
cadets.  Thus,  any  intra-organizational  competition  would  be  a  "synthetic"  one 
for  purposes  of  in5>rovement,  as  may  occur  in  multi-source  defense  contracts.®^ 

In  fact,  true  con^etition  should  be  a  "non-issue"  since  commissioning 
sources  are  integral  parts  of  DOD,  more  analogous  to  corporate  divisions .  As 
such,  the  culture  of  commissioning  may  best  be  served  by  benchmarking. 


24 


Benchmarking  supports  the  collection  of  evaluative  data  for  formative  purposes, 
and  its  use  by  commissioning  sources  extends  to  non-service  organizations  as 
well  as  service  counterparts.  In  areas  relating  to  officership,  benchmarking 
naturally  stems  from  other  commissioning  programs  in  the  US  and  other  nations; 
in  broader  leadership  training  it  may  include  even  more  organizations. 
Benchmarking  can  be  a  cooperative  effort  to  inqprove  commissioning  from  all 
sources .  What  is  needed  is  a  renewed  vision  of  DOD  corporateness  in 
commissioning  and  the  establishment  of  metrics  to  aid  benchmarking,  formative 
evaluation,  and  cooperation  throughout.  This  vision  and  these  metrics  must 
involve  customers,  planners,  and  decision  makers  at  all  levels.  The  issue 
remains  of  what  kinds  of  data  are  needed  (input-output,  subjective-objective, 
internal-external,  formative-svunmative)  and  over  what  span  it  should  be  applied. 

Regarding  input  or  output  data,  GAO  has  noted  that  the  service  academies' 
entrance  criteria  place  them  among  the  elite  of  the  nation’s  colleges  and 
universities.  For  exan^le,  in  1988  entrants  had  average  Scholastic  Aptitude 
Test  scores  ranging  from  564-588  (verbal)  and  642-668  (math)  con^ared  to 
national  averages  of  428  and  476,  respectively.  These  scores  were  paralleled  by 
student  records  showing  leadership  potential  and  athletic  aptitude. As  noted, 
though,  on  the  faculty-side  of  the  learning  input  equation  the  (3A0  questioned 
the  credentials  of  military  faculty.®^  Also,  in  1991,  the  GAO  termed  many 
output  indicators  of  program  effectiveness  "inadequate."  The  GAO  noted  academy 
graduates  "have  higher  retention  and  faster  career  progression  than  officers 
from  other  commissioning  sources,"  with  faster  promotion  and  higher  final 
rank.®®  The  GAO  concluded,  though,  that  personnel  policies  favoring  academy 
graduates  might  invalidate  such  measures — this  is  a  point  disputed  by  DOD.®® 
The  GAO  also  stated  that  "none  of  the  services  was  able  to  provide  much 
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objective  data  to  support  their  beliefs  that  their  prograitks  were  producing 
outstanding  officers" — here  DOD  concurred. Occasional  surveys  of  supervisory 
ratings  and  graduate  opinions  were  faulted  for  not  being  systematically 
conducted  or  con^arable  over  tlme.^^^^  Thus,  more  data  on  student  learning 
assessment,  program  output  tied  to  goals,  and  long-term  benefits  Is  needed. 

As  In  academia,  there  are  a  variety  of  metrics  available  to  commissioning 
sources  for  objective  evaluation.  Examinations,  despite  limitations,  remain 
standard  features,  though  USAFA  and  other  programs  may  look  more  at  other  types 
of  evaluation  (e.g.,  portfolios,  papers,  or  exercises)  . Job  performance, 
promotion  records,  ano  other  personnel  data  have  the  benefit  of  being  open  or 
quantifiable  to  an  extent;  however,  they  are  not  purely  objective.  Performance 
records  Include  narrative  evaluation  that  may  be  Inflated  for  promotional 
purposes.  As  noted  earlier,  job  records,  advancement  statistics,  or  other 
personnel  data  are  subject  to  interpretation  and  controversy.  As  such,  they 
are  not  good  metrics  from  an  idealized  TQM  standpoint — measures  that  are 
accepted  as  meaningful  by  the  customer (s),  understandable,  unambiguous,  and  able 
to  drive  appropriate  action. Developing  objective,  appropriate  metrics  has 
proven  to  be  an  ongoing  challenge  In  academia  and  military  educational  programs . 

As  TQM  depends  on  customer  perceptions,  subjective  Information  has  a 
special  place  In  educational  evaluation.  Subjective  information  that  Inpacts 
commissioning  programs  has  included  both  anecdotal  and  formal  feedback. USAFA 
graduates  are  cited  by  many  as  a  top  source  of  professionalism  and  conpetence. 
Alternately,  civilian  educators  working  with  USAFA  graduates  in  advanced 
programs  report  that  those  students  had  more  difficulty  orienting  to  their  work 
than  civilian  counterparts.^®^  They  may  also  be  seen  as  elitist,  less  mature, 
or  less  dedicated  to  continuous  learning.  It  has  been  suggested  that  a  lack  of 
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maturity  or  motivation  in  USAFA  graduates  might  be  linked  to  disciplinary 
problems. Similar  results  have  been  reported  for  other  academies.  West  Point 
graduates,  in  a  1987  survey,  fared  well  in  many  areas  except  traits  of  maturity, 
specific  job  knowledge  or  skills,  and  concern  for  their  troops. Quantifying 
such  views  while  avoiding  biases  or  stereotypes  is  difficult,  but  they  must  be 
considered  in  evaluating  and  in^roving  the  commissioning  process. 

There  also  are  dangers  in  too  much  focus  on  quantification,  especially  in 
mandated  metrics.  The  focus  in  TQM  is  for  internal  assessment  and  formative 
evaluation  while  driving  out  fear  of  management.  This  argues  that  the  main 
focus  for  commissioning  metrics  and  improvement  should  be  within  each  program. 
Yet  the  GAO  has  stated  that  "internal  reviews  cannot  substitute  for  independent, 
external  oversight."^®®  It  feels  that  without  such  oversight  "the  Congress  and 
the  DOD  cannot  determine  readily  how  much  academy  graduates  cost  or  what  the 
Nation  is  getting  for  its  investment."^®®  In  part  this  attitude  may  stem  from  a 
GAO  concern  that  external  recommendations  by  accreditation  officials,  visiting 
professors,  and  the  GAO  regarding  the  relative  experience  of  academy  faculties 
or  the  level  of  work  required  of  cadets  had  little  independent  impact  on  service 
academies .  Some  such  concerns  might  be  met  by  broader  internal  evaluation, 
with  more  common  assessment  among  commissioning  sources  and  with  data  suitable 
for  both  formative  and  svunmative  purposes.  It  is  the  task  of  program  leaders  to 
ensure  that  valid  data  linked  to  educational  goals  and  processes  is  collected 
and,  even  if  externally  prompted  or  summative  in  purpose,  is  used  formatively. 

In  responding  to  1991  GAO  recommendations  that  it  needed  broader  data  on 
quality  and  performance,  DOD  argued  that  rather  than  tasking  the  services  to 
perform  that  task  it  would  do  so  itself.  The  Defense  Manpower  Data  Center 
designed  a  customer-oriented  survey  to  measure  performance,  with  anticipated 
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validation  of  the  survey  instrument  to  begin  by  the  summer  of  1991.  This  effort 
was  related  to  prior  GAO  studies  of  the  ROTC  system,  and  it  was  expected  to 
yield  results  also  applicable  to  other  coimtdssioning  programs. The  survey  is 
still  undergoing  validation,  though,  and  has  not  been  applied. 

Within  the  Air  Force,  in  1960  the  Occupational  Measurement  Center  created 
a  survey  for  the  CEC.  This  survey  explored  skills  needed  by  all  new  officers, 
and  its  results  formed  the  basis  of  the  CEMU.  A  second  survey  in  1984  focused 
on  job  performance  to  refine  the  CEMU.  Another  survey,  first  planned  for  1992, 
should  be  conducted  this  year — it  was  developed  by  Air  Training  Command  with  the 
Occupational  Measurement  Center  to  survey  officers  on  active,  reserve,  and 
national  guard  duty  concerning  the  tasks  performed  by  new  officers  and  the 
knowledge  required  of  them.*^^^  Along  similar  lines,  USAFA's  Department  of 
Behavioral  Sciences  and  Leadership  has  developed  a  "behaviorally  anchored  rating 
scale"  that  can  help  evaluate  cadet  performance  more  guantifiably  in  key  areas 
related  to  military  performance  and  of ficership. This  survey  could  be  adapted 
to  monitor  officers  after  graduation  and  strengthen  the  evaluative  link  between 
commissioning  programs  and  career  performance.  Extending  such  data  throughout 
the  career  provides  a  better  evaluation  of  cost-benefits  and  long-term 
performance  while  providing  a  link  to  follow-on  professional  military  education. 

With  surveys  or  other  needed  metrics  there  is  merit  in  DOD-wide 
development  and  application.  This  would  require  all  commissioning  sources  to 
agree  on  the  qualities  sought  in  our  newly  commissioned  officers,  objectives  of 
the  various  commissioning  programs,  and  meaningful  metrics  at  different  levels. 
There  are  unique  aspects  and  resources  in  each  commissioning  program;  still,  the 
effort  expended  in  drawing  together  the  community  to  discuss  and  develop  such 
metrics  for  formative  as  well  as  summative  purposes  would  be  of  value  itself. 
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CHAPTER  V 


CONCLUSIONS 

This  paper  has  focused  from  broad  philosophy  to  specific  tools  in  one 
exploration  of  total-quality,  total-force  commissioning.  It  has  shown  that  many 
pressures  exist  for  evaluation  and  inq>rovement  of  DOD  commissioning  programs, 
pressures  internal  and  external  to  the  programs  themselves .  It  has  also  been 
seen  that  TQM  provides  a  way  to  meet  these  two  directional  pressures,  offering 
opportunities  for  improvement  and  integration. 

TQM,  meshed  with  traditional  concerns  for  educational  development  and 
tailored  to  commissioning  programs,  can  guide  continuous  in^rovement.  However, 
applying  TQM  principles  and  practices  highlights  issues  that  must  be  addressed. 
Among  these  are  defining  and  measuring  quality  for  broad  educational  and 
training  goals,  defining  multiple  customers  and  their  requirements  as  well  as 
multiple  products  and  their  priorities,  and  developing  meaningful  ways  to 
present  data  for  evaluation  to  senior  decision  makers.  Each  commissioning 
program  has  a  unique  nature  and  resources,  making  coitqparison  to  civilian  or 
functional  counterparts  difficult.  In  this  environment,  benchmarking  seems  more 
applicable  than  direct  con^etition,  either  with  civilian  institutions  or  with 
Interservlce  and  Intraservice  counterparts. 

In  application,  this  paper  has  focused  on  USAFA,  but  there  is  a  common 
mission  and  nature  to  commissioning  programs  as  they  generate  officers  for 
initial  service  through  education  and  training.  The  application  of  TQM  must  be 
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tailored  and  explored  by  commissioning  sources  just  as  it  is  being  tailored  and 
explored  by  US  business  and  government.  TQM  is  not  intended  to  stan^  out 
organizational  or  personal  individuality  in  the  name  of  standardization.  Rather 
it  frees  individuality  while  establishing  boundaries  and  measures  for  process 
in¥>rovement. In  a  similar  vein,  the  author  does  not  advocate  homogeneous 
commissioning.  Jointness  is  best  recognized  as  taking  advantage  of  the  unique 
capabilities  of  each  service;  this  holds  true  in  commissioning  programs. 

Services  can  retain  the  lead  in  evaluating  and  integrating  their  programs 
for  formative  development  while  supporting  other  summative  evaluation.  The  Air 
Force  has  one  mechanism  for  such  integration;  mechanisms  are  weaker  at  the  DOD 
level,  but  increased  coordination  has  been  promised  to  Congress.  Finding 
appropriate  metrics  remains  a  challenge  and  an  opportunity  in  this  process.  TQM 
favors  quantitative,  output-oriented,  internally  generated,  formative  metrics; 
however,  a  mix  of  these  with  qualitative,  input-oriented,  externally  mandated, 
and  summative  metrics  is  inevitable.  Program  leaders  should  try  to  participate 
in  developing  metrics,  even  mandated  ones,  and  to  ensure  that  the  mix  of  metrics 
ultimately  both  serves  decision  makers  and  fosters  program  improvement. 

This  paper,  in  focusing  from  broad  issues  to  specific  metrics,  highlighted 
the  current  use  of  periodic  surveys  and  the  difficulties  in  finding  other  needed 
metrics  for  ongoing  evaluation.  In  developing  such  metrics  basic  questions 
resurface  regarding  the  structure  and  possible  integration  of  our  education  and 
training  programs.  The  problem  is  complex  and  political,  and  this  paper  may 
have  provided  more  survey  than  solutions.  Still,  the  issues  reflected  here  are 
some  of  those  that  must  occupy  leaders,  planners,  and  senior  decision  makers  if 
we  are  to  have  programs  based  on  the  DOD  management  system  of  TQM  and  integrated 
in  a  total-force  environment  serving  the  individual  and  the  nation. 
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APPEKDIX  1 


MISSIONS  OF  CO^O^ISSIONING  PROGRAMS^^® 


SERVICE  ACADEMIES 

1.  Air  Force 

pre-1992:  "To  provide  instruction  and  experience  to  all  cadets  so  they  graduate 
with  the  knowledge,  character  and  motivation  essential  to  leadership  as  career 
officers  in  the  United  States  Air  Force." 

1992  proposal:  "To  develop  and  inspire  air  and  space  leaders  with  vision  for 
tomorrow. " 

2.  Navy 

"To  develop  midshipmen  morally,  mentally,  and  physically  and  to  imbue  them  with 
the  highest  ideals  of  duty,  honor,  and  loyalty  in  order  to  provide  graduates  who 
are  dedicated  to  a  career  of  naval  service  and  have  potential  for  future 
development  in  mind  and  character  to  assume  the  highest  responsibilities  of 
command,  citizenship,  and  government." 

3.  Army 

"To  educate  and  train  the  Corps  of  Cadets  so  that  each  graduate  shall  have  the 
attributes  essential  to  prof-ssional  growth  as  an  officer  of  the  Regular  Army, 
and  to  inspire  each  to  a  lifetime  of  service  to  the  nation." 


AIR  FORCE  RESERVE  OFFICER  TRAINING  CORPS 

"To  provide  instruction  and  experience  to  all  cadets  in  a  diversified  college  or 
university  environment  so  they  will  graduate  with  the  knowledge,  character  and 
motivation  essential  to  become  warrior  leaders  in  the  US  Air  Force." 


AIR  FORCE  OFFICER  TRAINING  SCHOOL 

"To  lead,  train,  motivate  and  commission  candidates  as  officers  in  response  to 
US  Air  Force,  Air  National  Guard,  and  US  Air  Force  Reserve  requirements." 
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APPENDIX  II 

W.  EDWARDS  DEMING'S  14  POINTS  FOR  TOTAL  QUALITY  MANAGEMENT^'' 

1.  Create  constancy  o£  purpose  toward  in^rovement  of  product  and  service,  with 
the  aim  to  stay  in  competitive  business  and  to  provide  jobs. 

2.  Adopt  the  new  philosophy.  Become  intolerant  of  comnonly  accepted  levels  of 
delays,  mistakes,  defective  materials,  and  defective  workmanship.  Do  not 
accept  bureaucratic  size  or  inertia  as  an  excuse. 

3.  Stop  depending  on  mass  inspection  to  achieve  quality.  Require,  instead, 
statistical  evidence  that  quality  is  built  into  the  process. 

4.  Stop  awarding  business  on  the  basis  of  its  price  tag  alone.  Find  meaningful 
measures  of  total  value  and  seek  to  build  long-term  relations  of  loyalty  and 
trust  with  suppliers. 

5.  Improve  constantly  and  forever  the  system  of  production  and  service.  Find 
problems — it  is  management's  job  to  work  continually  with  workers  on  the 
system  to  reduce  waste  and  improve  quality. 

6.  Institute  modern  methods  of  training  on  the  job. 

7 .  Institute  leadership  and  modern  methods  of  supervision  aimed  at  helping 
people  do  a  better  job. 

8.  Drive  out  fear,  so  that  everyone  may  work  effectively  for  the  company. 
Replace  external  motivators  with  internal  ones . 

9.  Break  down  barriers  between  departments  or  staff  areas.  Enhance  communi¬ 
cations  and  replace  destructive  competition  or  conflicting  goals  with 
teamwork  towards  the  common  goal . 

10.  Eliminate  arbitrary  numerical  goals,  posters,  and  slogans  for  the  workforce, 
or  asking  for  new  levels  of  productivity  without  providing  methods . 

11.  Eliminate  work  standards  that  prescribe  numerical  quotas.  Substitute  leader¬ 
ship  and  stress  quality  over  numbers. 

12.  Remove  barriers  that  stand  between  the  worker  (engineer,  manager,  etc.)  and 
the  right  to  pride  in  workmanship. 

13.  Institute  a  vigorous  program  of  education,  training,  and  self-iit^rovement  to 
include  TQM  philosophy  and  methods . 

14.  Create  a  structure  of  top  management  that  will  lead  by  exairple  while  making 
the  TQM  transformation  everyone’s  job,  every  day. 
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APPENDIX  III 


JOSEPH  M.  JURAN'S  10  STEPS  TO  QUALITY  IMPROVEMENT^ 


1.  Build  awareness  of  the  need  and  opportunity  for  improvement. 

2.  Set  goals  for  improvement. 

3.  Organize  to  reach  the  goals  (establish  a  quality  council,  identify  problems, 
select  projects,  appoint  teauns,  and  designate  facilitators) . 

4.  Provide  training. 

5.  Carry  out  projects  to  solve  problems. 

6.  Report  progress. 

7.  Give  recognition. 

8.  Communicate  results. 

9.  Keep  score. 

10.  Maintain  momentum  by  making  annual  in^rovement  part  of  the  regular  systems 
and  processes  of  the  company. 
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APPENDIX  IV 


PHILIP  B.  CROSBY'S  14  PRINCIPLES  OR  STEPS  TO  QUALITY  IMPROVEMENT^^ 


1.  Management  Coiranitment.  Make  it  clear  that  management  is  committed  to 
quality. 

2.  Quality  In^rovement  Teams.  Form  teams  with  representatives  from  each 
department. 

3.  Measurement.  Determine  where  current  and  potential  quality  problems  lie. 

4.  Cost  of  Quality.  Evaluate  the  cost  of  quality  and  explain  its  use  as  a 
management  tool. 

5.  Quality  Awareness.  Raise  the  quality  awareness  and  personal  concern  of  all 
employees . 

6.  Corrective  action.  Take  actions  to  correct  problems  identified  in  previous 
steps . 

7.  Zero  Defects  Planning.  Establish  a  committee  for  the  zero  defects  programs. 
[T^ply  zero  defects  concepts  as  a  common  goal,  to  be  sought  by  continuous 
in^rovement,  rather  than  as  a  motivational  and  assessment  program. ) 

8.  En^loyee  Education.  Train  supervisors  and  all  employees  to  actively  carry 
out  their  part  of  the  quality  in^rovement  program. 

9.  Zero  Defects  Day.  Hold  a  "zero  defects  day"  to  let  all  enployees  realize 
that  there  has  been  a  change. 

10.  Goal  Setting.  Encourage  individuals  to  establish  improvement  goals  for 
themselves  and  their  groups. 

11.  Error  Cause  Removal.  Encourage  en^loyees  to  communicate  to  management  the 
obstacles  they  face  in  attaining  their  improvement  goals. 

12.  Recognition.  Recognize  and  appreciate  those  who  participate. 

13.  Quality  Councils.  Establish  quality  councils  to  communicate  on  a  regular 
basis . 

14.  Do  It  All  Over  Again.  En^hasize  that  the  quality  improvement  process  never 
ends . 
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APPENDIX  V 


WILLIAM  E.  CONWAY'S  6  TOOLS  FOR  QUALITY  IMPROVEMENT^^O 


1.  Human  relations  skills — the  responsibility  of  management  to  create  at  every 
level,  among  all  employees,  the  motivation  and  training  to  make  necessary 
inprovements  in  the  organization. 

2.  Statistical  surveys — the  gathering  of  data  about  customers  (internal  as  well 
as  external),  employees,  technology  and  equipment,  to  be  used  as  a  measure 
for  future  progress  and  to  identify  what  needs  to  be  done. 

3.  Single  statistical  techniques — clear  charts  and  diagrams  that  help  identify 
problems,  track  work  flow,  gauge  progress,  and  indicate  solutions. 

4.  Statistical  process  control — the  statistical  charting  of  a  process,  whether 
manufacturing  or  non-manufacturing,  to  help  identify  and  reduce  variation. 

5.  Imagineering — a  key  concept  in  problem  solving,  involving  the  visualization 
of  a  process,  procedure,  or  operation  with  all  waste  eliminated. 

6.  Industrial  engineering — common  techniques  of  pacing,  work  simplification, 
methods  analysis,  plant  layout  and  material  handling  to  achieve  inprovement. 
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APPENDIX  VI 


MICHAEL  E.  MILAKOVICH'S  ELEMENTS  OF  TQM  IN  PUBLIC  SERVICE^^l 


1.  Customer  satisfaction  is  the  primary  goal  and  ultimate  measure  of  quality  in 
any  public  service  organization. 

2.  The  definition  of  customer  is  broadened  to  include  both  those  internal  to 
the  organization  (for  ex2ut^le,  employees  in  other  departments)  and  those 
external  to  the  organization  (vendors,  taxpayers,  contractors,  regulators, 
suppliers) . 

3.  Everyone  must  share  a  common  vision  of  the  mission  of  the  organization  based 
on  extended  customer  requirements. 

4.  Senior  elected  and  appointed  leaders  must  communicate  a  long-term  commitment 
to  all  customers,  reward  teamworlc,  and  encourage  process  in5>rovement  efforts 
at  all  levels. 

5.  Expanded  training  and  self-improvement  opportunities  in  leadership  s)cills 
must  be  offered  to  meet  or  exceed  valid  customer  requirements. 

6.  Individual  involvement  must  be  ensured  by  establishing  process  in^rovement 
teams . 

7.  En^loyee  loyalty,  trust,  and  teamwork  must  be  recognized,  supported,  and 
acknowledged. 

8.  Fear  of  change  must  be  eliminated,  and  other  barriers  to  the  development  of 
pride  in  service  must  be  removed. 

9.  Everyone  must  be  provided  with  the  tools  and  training  needed  to  function  in 
accordance  with  extended  customer  requirements. 

10.  Senior  elected  and  appointed  officials  must  make  the  necessary  changes  in 
public  organizations  for  successful  implementation  of  the  preceding  goals  to 
become  possible. 
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APPENDIX  VII 


ONE  COMPARISON  OF  CLASSIC  EDUCATIONAL  ASSESSMENT  AND  T<»4^22 


Assessment  Emphasizes 

Goals 

Iterative 

Talk  to  Students 

Collaboration 

Done  by  Faculty 

Focus  on  Outcomes 
Data-driven 


TQM  Emphasizes 

Vision 

Continuous 

Close  to  Customers 

Teamwork 

Faculty  and  Students 
Responsible  for  Process 

Start  with  Process 

Data-driven 


NOTES ; 

Both  educational  assessment  and  TQM  are  shown  as  data-driven. 

Chaffee  and  Ewell,  in  comments  on  this  con^arison,  agree  that  TQM  offers 
a  broader  scope  than  traditional  assessment. 

Elsewhere,  Ewell  has  noted  other  parallels  in  TQM  and  assessment 
practices  such  as  (flowcharting  -  curricular  mapping)  or  (san^ling  - 
portfolio  evaluation) 
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APPENDIX  VIII 


SOME  AREAS  USED  IN  ASSESSMENT  AND  EVALUATION^^^ 


Program  costs  (total,  per  student,  by  program,  etc.) 

Efficiency  of  administrative  support  or  procedures 

Size  or  nature  of  facilities  (library,  labs,  classrooms,  gyms,  etc.) 
Availability  of  con^uters,  media,  or  other  equipment 
Student-teacher  ratio 
Lab-lecture  teaching  format  ratio 

Population  characteristics:  class  size,  sex,  race,  minority  representation, 
regional  origins,  etc. 

Incoming  standardized  test  scores  (SAT,  ACT,  etc.)  or  other  achievements 
Number  of  faculty  PhDs,  Nobel  laureates,  or  endowed  chairs 
Faculty  research  programs,  grants  and  funding  levels 
Faculty  research  results  and  renown  or  awards 

Number  of  faculty  publications  and  citations  of  their  publications  by  other 
scholars 

Instructor  characteristics:  who  teaches  (teaching  assistants  or  professors), 
training  and  experience  levels,  time  in  class,  availability  for  tutoring 
Teaching  style  and  technique  of  instructors 

Responsiveness  of  teachers  to  students  (extra  instruction,  time  ta)cen  to  give 
feedback  on  assignments,  etc.) 

Curriculum  design  and  course  offerings 
Course  and  graduation  requirements 

Course  materials  (tests,  handouts,  etc.),  texts,  and  reading  assignments 
Existence  of  collocated  undergraduate  and  graduate  programs 
Cooperative  or  collaborative  progreuns  with  other  institutions 
Niunber  and  type  of  enrichment  programs  offered 

Involvement  and  levels  of  performance  of  students  in  enrichment  programs 
(research,  athletics,  social,  community,  etc.) 

Number  of  returning  freshmen  (an  interim  measure) 

Grades  and  other  performance  measures  in  each  course 

Student  portfolios  or  other  coit^ilations  of  work  during  student  career 
National  honors  and  recognition  of  students 

Performance  and  relative  placement  of  students  in  scholastic  con^>etitions 
Student  learning-related  behaviors,  disciplinary  problems,  etc. 

Specific  tests  of  behaviors:  critical  thinking,  attitudes,  values,  etc. 

Entry,  interim,  exit,  post-exit,  and  special  surveys  or  interviews  (students, 
faculty,  alumni,  supervisors,  external  experts,  etc.) 

Anecdotal  information  and  feedback 

Nximber  of  graduates,  percent  con^letion,  and  time-to-completion 
Niunber  and  types  of  degrees  attained  by  students,  grades  at  completion, 
disciplinary  and  non-disciplinary  majors  obtained,  etc. 

Standardized  exit  test  scores  (in  capstone  courses,  on  GRE,  etc.) 

National  awards  and  scholarships  earned  by  graduating  students 
Levels  and  prestige  of  graduate  school  acceptances  for  students 
Job  placement  and  time-to-placement,  level  of  job  placement,  and  initial 
salaries  of  graduates 

Long-term  job  promotion,  awards,  and  professional  achievement  of  graduates 
Institutional  or  departmental  rankings  and  accreditation,  regional  or  national 
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